Clinician perspectives on how situational context and augmented intelligence design features impact perceived usefulness of sepsis prediction scores embedded within a simulated electronic health record.

OBJECTIVE: Obtain clinicians' perspectives on early warning scores (EWS) use within context of clinical cases. MATERIAL AND METHODS: We developed cases mimicking sepsis situations. De-identified data, synthesized physician notes, and EWS representing deterioration risk were displayed in a simulated EHR for analysis. Twelve clinicians participated in semi-structured interviews to ascertain perspectives across four domains: (1) Familiarity with and understanding of artificial intelligence (AI), prediction models and risk scores; (2) Clinical reasoning processes; (3) Impression and response to EWS; and (4) Interface design. Transcripts were coded and analyzed using content and thematic analysis. RESULTS: Analysis revealed clinicians have experience but limited AI and prediction/risk modeling understanding. Case assessments were primarily based on clinical data. EWS went unmentioned during initial case analysis; although when prompted to comment on it, they discussed it in subsequent cases. Clinicians were unsure how to interpret or apply the EWS, and desired evidence on its derivation and validation. Design recommendations centered around EWS display in multi-patient lists for triage, and EWS trends within the patient record. Themes included a "Trust but Verify" approach to AI and early warning information, dichotomy that EWS is helpful for triage yet has disproportional signal-to-high noise ratio, and action driven by clinical judgment, not the EWS. CONCLUSIONS: Clinicians were unsure of how to apply EWS, acted on clinical data, desired score composition and validation information, and felt EWS was most useful when embedded in multi-patient views. Systems providing interactive visualization may facilitate EWS transparency and increase confidence in AI-generated information.

Accomplished women leaders in informatics: insights about successful careers.

We sought to learn from the experiences of women leaders in informatics by interviewing women in Informatics leadership roles. Participants reported career challenges, how they built confidence, advice to their younger selves, and suggestions for attracting and retaining additional women. Respondents were 16 women in leadership roles in academia (n = 9) and industry (n = 7). We conducted a thematic analysis revealing: (1) careers in informatics are serendipitous and nurtured by supportive communities, (2) challenges in leadership were profoundly related to gender issues, (3) "Big wins" in informatics careers were about making a difference, and (4) women leaders highlighted resilience, excellence, and personal authenticity as important for future women leaders. Sexism is undeniably present, although not all participants reported overt gender barriers. Confidence and authenticity in leadership point to the value offered by individual leaders. The next step is to continue to foster an informatics culture that encourages authenticity across the gender spectrum.

Model depicting aspects of audit and feedback that impact physicians' acceptance of clinical performance feedback.

BACKGROUND: Audit and feedback (A&F) is a strategy that has been used in various disciplines for performance and quality improvement. There is limited research regarding medical professionals' acceptance of clinical-performance feedback and whether feedback impacts clinical practice. The objectives of our research were to (1) investigate aspects of A&F that impact physicians' acceptance of performance feedback; (2) determine actions physicians take when receiving feedback; and (3) determine if feedback impacts physicians' patient-management behavior. METHODS: In this qualitative study, we employed grounded theory methods to perform a secondary analysis of semi-structured interviews with 12 VA primary care physicians. We analyzed a subset of interview questions from the primary study, which aimed to determine how providers of high, low and moderately performing VA medical centers use performance feedback to maintain and improve quality of care, and determine perceived utility of performance feedback. RESULTS: Based on the themes emergent from our analysis and their observed relationships, we developed a model depicting aspects of the A&F process that impact feedback acceptance and physicians' patient-management behavior. The model is comprised of three core components - Reaction, Action and Impact - and depicts elements associated with feedback recipients' reaction to feedback, action taken when feedback is received, and physicians modifying their patient-management behavior. Feedback characteristics, the environment, external locus-of-control components, core values, emotion and the assessment process induce or deter reaction, action and impact. Feedback characteristics (content and timeliness), and the procedural justice of the assessment process (unjust penalties) impact feedback acceptance. External locus-of-control elements (financial incentives, competition), the environment (patient volume, time constraints) and emotion impact patient-management behavior. Receiving feedback generated intense emotion within physicians. The underlying source of the emotion was the assessment process, not the feedback. The emotional response impacted acceptance, impelled action or inaction, and impacted patient-management behavior. Emotion intensity was associated with type of action taken (defensive, proactive, retroactive). CONCLUSIONS: Feedback acceptance and impact have as much to do with the performance assessment process as it does the feedback. In order to enhance feedback acceptance and the impact of feedback, developers of clinical performance systems and feedback interventions should consider multiple design elements.

Using voluntary reports from physicians to learn from diagnostic errors in emergency medicine.

OBJECTIVES: Diagnostic errors are common in the emergency department (ED), but few studies have comprehensively evaluated their types and origins. We analysed incidents reported by ED physicians to determine disease conditions, contributory factors and patient harm associated with ED-related diagnostic errors. METHODS: Between 1 March 2009 and 31 December 2013, ED physicians reported 509 incidents using a department-specific voluntary incident-reporting system that we implemented at two large academic hospital-affiliated EDs. For this study, we analysed 209 incidents related to diagnosis. A quality assurance team led by an ED physician champion reviewed each incident and interviewed physicians when necessary to confirm the presence/absence of diagnostic error and to determine the contributory factors. We generated descriptive statistics quantifying disease conditions involved, contributory factors and patient harm from errors. RESULTS: Among the 209 incidents, we identified 214 diagnostic errors associated with 65 unique diseases/conditions, including sepsis (9.6%), acute coronary syndrome (9.1%), fractures (8.6%) and vascular injuries (8.6%). Contributory factors included cognitive (n=317), system related (n=192) and non-remedial (n=106). Cognitive factors included faulty information verification (41.3%) and faulty information processing (30.6%) whereas system factors included high workload (34.4%) and inefficient ED processes (40.1%). Non-remediable factors included atypical presentation (31.3%) and the patients' inability to provide a history (31.3%). Most errors (75%) involved multiple factors. Major harm was associated with 34/209 (16.3%) of reported incidents. CONCLUSIONS: Most diagnostic errors in ED appeared to relate to common disease conditions. While sustaining diagnostic error reporting programmes might be challenging, our analysis reveals the potential value of such systems in identifying targets for improving patient safety in the ED.

Patient-initiated second opinions: systematic review of characteristics and impact on diagnosis, treatment, and satisfaction.

The impact of second opinions on diagnosis in radiology and pathology is well documented; however, the value of patient-initiated second opinions for diagnosis and treatment in general medical practice is unknown. We conducted a systematic review of patient-initiated second opinions to assess their impact on clinical outcomes and patient satisfaction and to determine characteristics and motivating factors of patients who seek a second opinion. We searched PubMed, EMBASE, Cochrane, and Academic OneFile databases using Medical Subject Headings (MeSH) indexes and keyword searches. Search terms included referral and consultation, patient-initiated, patient preference, patient participation, second opinion, second review, and diagnosis. Multiple reviewers screened abstracts and articles to determine eligibility and extract data. We assessed risk of bias using the Cochrane Risk of Bias Tool and rated study quality using Cochrane's GRADE (Grades of Recommendation, Assessment, Development and Evaluation) approach. We screened 1342 abstracts and reviewed full text of 41 articles, identifying 7 articles that reported clinical agreement data and 10 that discussed patient characteristics, motivation, and satisfaction. We found that a second opinion typically confirms the original diagnosis or treatment regimen but that 90% of patients with poorly defined conditions remain undiagnosed. However, 10% to 62% of second opinions yield a major change in the diagnosis, treatment, or prognosis. A larger fraction of patients receive different advice on treatment than on diagnosis. Factors motivating a second opinion include diagnosis or treatment confirmation, dissatisfaction with a consultation, desire for more information, persistent symptoms, or treatment complications. Patients generally believed that second opinions were valuable. Second opinions can result in diagnostic and treatment differences. The literature on patient-initiated second opinions is limited, and the accuracy of the second opinion through follow-up is generally unknown. Standardized methods and outcome measures are needed to determine the value of second opinions, and the potential of second opinions to reduce diagnostic errors merits more rigorous evaluation.

Physicians' diagnostic accuracy, confidence, and resource requests: a vignette study.

IMPORTANCE: Little is known about the relationship between physicians' diagnostic accuracy and their confidence in that accuracy. OBJECTIVE: To evaluate how physicians' diagnostic calibration, defined as the relationship between diagnostic accuracy and confidence in that accuracy, changes with evolution of the diagnostic process and with increasing diagnostic difficulty of clinical case vignettes. DESIGN, SETTING, AND PARTICIPANTS: We recruited general internists from an online physician community and asked them to diagnose 4 previously validated case vignettes of variable difficulty (2 easier; 2 more difficult). Cases were presented in a web-based format and divided into 4 sequential phases simulating diagnosis evolution: history, physical examination, general diagnostic testing data, and definitive diagnostic testing. After each phase, physicians recorded 1 to 3 differential diagnoses and corresponding judgments of confidence. Before being presented with definitive diagnostic data, physicians were asked to identify additional resources they would require to diagnose each case (ie, additional tests, second opinions, curbside consultations, referrals, and reference materials). MAIN OUTCOMES AND MEASURES: Diagnostic accuracy (scored as 0 or 1), confidence in diagnostic accuracy (on a scale of 0-10), diagnostic calibration, and whether additional resources were requested (no or yes). RESULTS: A total of 118 physicians with broad geographical representation within the United States correctly diagnosed 55.3% of easier and 5.8% of more difficult cases (P < .001). Despite a large difference in diagnostic accuracy between easier and more difficult cases, the difference in confidence was relatively small (7.2 vs 6.4 out of 10, for easier and more difficult cases, respectively) (P < .001) and likely clinically insignificant. Overall, diagnostic calibration was worse for more difficult cases (P < .001) and characterized by overconfidence in accuracy. Higher confidence was related to decreased requests for additional diagnostic tests (P = .01); higher case difficulty was related to more requests for additional reference materials (P = .01). CONCLUSIONS AND RELEVANCE: Our study suggests that physicians' level of confidence may be relatively insensitive to both diagnostic accuracy and case difficulty. This mismatch might prevent physicians from reexamining difficult cases where their diagnosis may be incorrect.

Development and validation of an algorithm to recalibrate mental models and reduce diagnostic errors associated with catheter-associated bacteriuria.

BACKGROUND: Overtreatment of catheter-associated bacteriuria is a quality and safety problem, despite the availability of evidence-based guidelines. Little is known about how guidelines-based knowledge is integrated into clinicians' mental models for diagnosing catheter-associated urinary tract infection (CA-UTI). The objectives of this research were to better understand clinicians' mental models for CA-UTI, and to develop and validate an algorithm to improve diagnostic accuracy for CA-UTI. METHODS: We conducted two phases of this research project. In phase one, 10 clinicians assessed and diagnosed four patient cases of catheter associated bacteriuria (n= 40 total cases). We assessed the clinical cues used when diagnosing these cases to determine if the mental models were IDSA guideline compliant. In phase two, we developed a diagnostic algorithm derived from the IDSA guidelines. IDSA guideline authors and non-expert clinicians evaluated the algorithm for content and face validity. In order to determine if diagnostic accuracy improved using the algorithm, we had experts and non-experts diagnose 71 cases of bacteriuria. RESULTS: Only 21 (53%) diagnoses made by clinicians without the algorithm were guidelines-concordant with fair inter-rater reliability between clinicians (Fleiss' kappa = 0.35, 95% Confidence Intervals (CIs) = 0.21 and 0.50). Evidence suggests that clinicians' mental models are inappropriately constructed in that clinicians endorsed guidelines-discordant cues as influential in their decision-making: pyuria, systemic leukocytosis, organism type and number, weakness, and elderly or frail patient. Using the algorithm, inter-rater reliability between the expert and each non-expert was substantial (Cohen's kappa = 0.72, 95% CIs = 0.52 and 0.93 between the expert and non-expert #1 and 0.80, 95% CIs = 0.61 and 0.99 between the expert and non-expert #2). CONCLUSIONS: Diagnostic errors occur when clinicians' mental models for catheter-associated bacteriuria include cues that are guidelines-discordant for CA-UTI. The understanding we gained of clinicians' mental models, especially diagnostic errors, and the algorithm developed to address these errors will inform interventions to improve the accuracy and reliability of CA-UTI diagnoses.

Cognitive interventions to reduce diagnostic error: a narrative review.

BACKGROUND: Errors in clinical reasoning occur in most cases in which the diagnosis is missed, delayed or wrong. The goal of this review was to identify interventions that might reduce the likelihood of these cognitive errors. DESIGN: We searched PubMed and other medical and non-medical databases and identified additional literature through references from the initial data set and suggestions from subject matter experts. Articles were included if they either suggested a possible intervention or formally evaluated an intervention and excluded if they focused solely on improving diagnostic tests or provider satisfaction. RESULTS: We identified 141 articles for full review, 42 reporting tested interventions to reduce the likelihood of cognitive errors, 100 containing suggestions, and one article with both suggested and tested interventions. Articles were classified into three categories: (1) Interventions to improve knowledge and experience, such as simulation-based training, improved feedback and education focused on a single disease; (2) Interventions to improve clinical reasoning and decision-making skills, such as reflective practice and active metacognitive review; and (3) Interventions that provide cognitive 'help' that included use of electronic records and integrated decision support, informaticians and facilitating access to information, second opinions and specialists. CONCLUSIONS: We identified a wide range of possible approaches to reduce cognitive errors in diagnosis. Not all the suggestions have been tested, and of those that have, the evaluations typically involved trainees in artificial settings, making it difficult to extrapolate the results to actual practice. Future progress in this area will require methodological refinements in outcome evaluation and rigorously evaluating interventions already suggested, many of which are well conceptualised and widely endorsed.

Effect of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths.

OBJECTIVES: Determine effects of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths. Determine if limited enforcement in a medical tutoring system inhibits students from learning the optimal and most efficient solution path. Describe the type of deviations from the optimal solution path that occur during tutoring, and how these deviations change over time. Determine if the size of the problem-space (domain scope), has an effect on learning gains when using a tutor with limited enforcement. METHODS: Analyzed data mined from 44 pathology residents using SlideTutor-a Medical Intelligent Tutoring System in Dermatopathology that teaches histopathologic diagnosis and reporting skills based on commonly used diagnostic algorithms. Two subdomains were included in the study representing sub-algorithms of different sizes and complexities. Effects of the tutoring system on student errors, goal states and solution paths were determined. RESULTS: Students gradually increase the frequency of steps that match the tutoring system's expectation of expert performance. Frequency of errors gradually declines in all categories of error significance. Student performance frequently differs from the tutor-defined optimal path. However, as students continue to be tutored, they approach the optimal solution path. Performance in both subdomains was similar for both errors and goal differences. However, the rate at which students progress toward the optimal solution path differs between the two domains. Tutoring in superficial perivascular dermatitis, the larger and more complex domain was associated with a slower rate of approximation towards the optimal solution path. CONCLUSIONS: Students benefit from a limited-enforcement tutoring system that leverages diagnostic algorithms but does not prevent alternative strategies. Even with limited enforcement, students converge toward the optimal solution path.

Assessing the use of cognitive heuristic representativeness in clinical reasoning.

We performed a pilot study to investigate use of the cognitive heuristic Representativeness in clinical reasoning. We tested a set of tasks and assessments to determine whether subjects used the heuristics in reasoning, to obtain initial frequencies of heuristic use and related cognitive errors, and to collect cognitive process data using think-aloud techniques. The study investigates two aspects of the Representativeness heuristic - judging by perceived frequency and representativeness as causal beliefs. Results show that subjects apply both aspects of the heuristic during reasoning, and make errors related to misapplication of these heuristics. Subjects in this study rarely used base rates, showed significant variability in their recall of base rates, demonstrated limited ability to use provided base rates, and favored causal data in diagnosis. We conclude that the tasks and assessments we have developed provide a suitable test-bed to study the cognitive processes underlying heuristic errors.